Apparatus, method, and program for detecting object from image

ABSTRACT

An image processing apparatus includes a detection unit configured to scan an input image and each of images at different resolutions, which are generated from the input image, by a predetermined-sized window to detect an object in the image, a storage unit configured to store a detection result of the detection unit, and a control unit configured to, if there is no free space in the storage unit to store a new detection result of the detection unit, store the new detection result instead of a detection result of an image at higher resolution than resolution of an image from which the new detection result is acquired.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus, a method, and a programfor detecting an object from an image.

2. Description of the Related Art

A technique for detecting a specific object pattern from an image isextremely useful and can be used for detection of a human face, forexample. This technique can be used in many fields such asteleconferences, man-machine interfaces, security, monitoring systemsfor tracking a human face, and image compression. Various methods torealize this technique for detecting a face in an image are mentioned in“Detecting Faces in Images: A Survey” written by M. H. Yang, D. J.Kriegman, and N. Ahuja, published in Institute of Electrical andElectronics Engineers (IEEE) Transactions on Pattern Analysis andMachine Intelligence (Trans. on PAMI), Volume 24, Number 1, Pages 34 to58, issued in January, 2002.

Especially, a method based on the Boosting algorithm proposed by Violaet al., which is discussed in “Robust Real-time Object Detection”written by P. Viola and M. Jones, published in Proceedings of IEEEWorkshop Statistical and Computational Theories of Vision (SCTV), issuedin July, 2001, is widely utilized in the research of the face detectiondue to its excellent execution speed and high accuracy of detectionrate. The method based on the Boosting algorithm proposed by Viola etal. can speed up the face detection by connecting substantiallyidentical small classification units (weak classifiers) in series andstopping detection processing when it is determined that the targetregion is not a face region in the middle of the detection processing toomit subsequent detection processing.

The size of a window image, which is a partial image referred to by theweak classifiers, is determined relative to a detection target having acertain presumed size. However, the size of a detection target in anactual image may continuously vary (for example, the size of thedetection target continuously varies when a digital camera capturesimages of the object while changing a distance therebetween). Therefore,it is desirable to design an actual classifier (make a classifier learn)so as to be capable of accepting a change in the size of a detectiontarget to some degree. However, for designing a classifier so as to beable to accept a change in the size, it is extremely difficult torealize a classifier that can detect a target even when, for example, adetection window contains only a part of the detection target becausethe detection target is extremely large. Similarly, it is also extremelydifficult to realize a classifier that can detect a target when thedetection target is extremely small relative to a detection window.

To solve this problem, there are two possible approaches. One approachis to increase the window size (also enlarge, for example, the referencearea of the weak classifiers accordingly). The other approach is to,while fixing the window size, change (especially, reduce) themagnification ratio of an input image and set equally-sized windows tothe several kinds of generated images with their magnification ratioschanged (reduced) to detect a target by the classifier. In eithermethod, it is difficult to continuously enlarge the window or reduce theimage, so that it is desirable that the classifier can accept a changein the size to some degree after all. Further, similarly, it is alsodesirable that the classifier can accept changes in the position and theshape to some degree.

However, the classifier capable of accepting changes in the size and theposition tends to provide a plurality of detection determinations aroundactually one detection target. This is because the position of thetarget may horizontally or vertically move in the image or the size maybe different as described above (due to the change in the window size orthe reduction of the image size). Therefore, integration processing,which integrates the plurality of detection results to output onedetection result for one detection target, is necessary. The integrationprocessing is a kind of clustering processing for clustering detectionresults distributed in a space in which the axes represent thehorizontal and vertical positions and the size. Alternatively, theintegration processing may be performed by using a space additionallyhaving, for example, an axis representing reliability (likelihood) of adetection result. Since the integration processing is necessary, it isconsidered that each of the above-described detection results by theclassifiers is an intermediate result, and it is desirable to configurea system which can store these intermediate results once and thenperform the integration processing thereon.

As described above, it is necessary to once store the detection resultsof the classifiers, which are intermediate results, but an issue here isto estimate how large storage area to be prepared therefor. However, howmany detection results are actually output from an image that is adetection target cannot be determined before actual execution of thedetection processing. A theoretical upper limit value is the number (N)of windows settable in that image, but in many cases, the number ofdetection results from actual detection applied to an image isapproximately 1 to 2% of N. Therefore, it is inefficient to prepare alarge storage area capable of storing detection results corresponding tothe theoretical upper limit value.

The size of the storage area may be not an important matter when thedetection processing is realized by, for example, software running on apersonal computer (PC), since a relatively high-capacity memory can beprepared in this case. However, in a case where detection processing isrealized on software or hardware embedded in a device, it is desirableto realize the detection processing with use of a predetermined (fixed)small storage capacity. However, performing the detection processingusing the predetermined small storage capacity may result in a shortageof the storage area exceptionally depending on the content of an image.In this case, data overflowing the storage capacity, among all detectionresults, should be discarded, but this leads to a problem of determiningwhich and how detection result should be discarded.

SUMMARY OF THE INVENTION

The present invention is directed to realizing object detectionprocessing with a small storage capacity.

According to an aspect of the present invention, an image processingapparatus includes a detection unit configured to scan an input imageand each of images at different resolutions, which are generated fromthe input image, by a predetermined-sized window to detect an object inthe image, a storage unit configured to store a detection result of thedetection unit, and a control unit configured to, if there is no freespace in the storage unit to store a new detection result of thedetection unit, store the new detection result instead of a detectionresult of an image at higher resolution than resolution of an image fromwhich the new detection result is acquired.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 illustrates an example of a hardware configuration of an imageprocessing apparatus according to a first exemplary embodiment of thepresent invention.

FIG. 2 illustrates an algorithm of face detection processing developedby applying the Boosting algorithm.

FIG. 3 illustrates processing for detecting a target by applying adetection window to an image.

FIG. 4 illustrates states in which a size of a face as a detectiontarget varies.

FIG. 5 illustrates a processing procedure of detection processing.

FIG. 6 illustrates that the detection algorithm returns a plurality ofdetection results for a single object.

FIG. 7 illustrates a state in which a plurality of matching windows isassociated with one another in consideration of magnification ratios ofimages.

FIG. 8 illustrates an image containing faces having various sizes.

FIG. 9 illustrates an example of intermediate results of detection forthe respective objects.

FIG. 10 illustrates processing applied to the image illustrated in FIG.8.

FIGS. 11A to 11C each illustrate a state of an intermediate resultstorage unit.

FIGS. 12A and 12B each illustrate an example of a detection result.

FIG. 13 is a flowchart illustrating processing for storing intermediateresults by circularly using intermediate result storage areas accordingto the first exemplary embodiment.

FIG. 14 illustrates an example of the hardware configuration of theimage processing apparatus according to the first exemplary embodiment.

FIG. 15 is a flowchart illustrating processing for storing intermediateresults by circularly using the intermediate result storage areasaccording to the first exemplary embodiment.

FIG. 16 illustrates an example of the hardware configuration of theimage processing apparatus according to the first exemplary embodiment.

FIG. 17 illustrates an example of a hardware configuration of an imageprocessing apparatus according to a third exemplary embodiment.

FIG. 18 is a flowchart illustrating processing for storing intermediateresults in buffers.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

FIG. 1 illustrates an example of a hardware configuration of an imageprocessing apparatus according to a first exemplary embodiment. An imagestorage unit 101 stores image data which is a processing target. Theimage storage unit 101 is constituted by, for example, a memory.Alternatively, the image storage unit 101 may be constituted by, forexample, a resister or a disk apparatus. A detection unit 102 performsdetection processing. An example of the detection processing will bedescribed below with reference to FIG. 2. The detection unit 102 outputsa detection result to an intermediate result storage unit 103. Further,the detection unit 102 instructs an integration processing unit 104 tostart integration processing, when the detection unit 102 detectscompletion of processing one image. The intermediate result storage unit103 stores the output of the detection unit 102. The intermediate resultstorage unit 103 is constituted by, for example, a memory.Alternatively, the intermediate result storage unit 103 may beconstituted by, for example, a resister or a disk apparatus.

The integration processing unit 104 integrates the intermediate resultsstored in the intermediate result storage unit 103, and outputs anintegration result thereof to a result storage unit 105. The integrationprocessing unit 104 operates according to the instruction of thedetection unit 102. The result storage unit 105 stores the output of theintegration processing unit 104. The result storage unit 105 isconstituted by, for example, a memory. Alternatively, the result storageunit 105 may be constituted by, for example, a resister or a diskapparatus. An image magnification varying unit 110 changes (at leastreduces) a magnification ratio of an image stored in the image storageunit 101, and outputs the image with its magnification ratio changed. Asillustrated in FIG. 1, the detection unit 102 is configured to be ableto read the image with its magnification ratio changed. The imagemagnification varying unit 110 is also configured to output the imagewith its magnification ratio changed to the image storage unit 101.Besides these units, various other units may be added to the imageprocessing unit of the present exemplary embodiment, but the descriptionthereof will be omitted herein since they are not main features of thepresent exemplary embodiment.

First, the detection processing by the detection unit 102 will bedescribed. The present exemplary embodiment will be described based onan example in which a detection target is set to a human face. Anexample of the algorithm related to the detection processing will bedescribed with reference to FIG. 2. The face detection processingdeveloped by applying the Boosting algorithm will be described by way ofexample. FIG. 2 illustrates the algorithm of the face detectionprocessing developed by applying the Boosting algorithm.

A partial image 201 is a partial region of input image data. The partialimage 201 is an image within a predetermined window set on the inputimage data, and includes an image region referred to by weak classifiers210 to 250, which will be described below. Hereinafter, the partialimage 201 will be also referred to as a “window image”. The weakclassifier 210 refers to a part or all of pixel values in the windowimage 201, performs predetermined calculation on the pixel values, anddetermines whether the window image 201 contains an object as adetection target (in the present example, a human face region), from thecalculation result. Parameters such as a reference position and range ofthe predetermined window image 201 and a coefficient used in thecalculation are determined by machine learning prior to the actualdetection processing.

The weak classifiers 211 to 250 also perform the calculation processingsimilar to the weak classifier 210. However, it is desirable that theweak classifiers 211 to 250 are configured in such a manner that theparameters such as the reference position and range of the window image201 and the coefficient used in the calculation are different from thoseof the weak classifier 210. The weak classifiers 211 to 250 can beconfigured in such a manner that the parameters such as the referenceposition and range of the window image 201 and the coefficient used inthe calculation are different from one another. In other words, the weakclassifiers 210 to 250 perform substantially identical processes, exceptfor the differences in the parameters such as the reference position andrange of the window image 201 and the coefficient used in thecalculation. Therefore, if the reference position and range of thewindow image 201, the coefficient, and the like used in the calculationare separated as parameters of the processing, the weak classifiers 210to 250 can be considered as identical processes using differentparameters.

It is assumed that N represents the number of the weak classifiersincluding the weak classifiers 210 to 250. Each of N pieces of the weakclassifiers determines PASS or NG according to the internal calculationresult, and these weak classifier processes are connected in series asillustrated in FIG. 2 to construct determination processing. Theprocessing is started from the 0-th weak classifier 210. If thedetermination result by the 0-th weak classifier 210 is PASS, thedetermination processing proceeds to the next weak classifier 211. Theprocessing is performed in this way. If the final weak classifier,namely the weak classifier 250 also determines PASS as the determinationresult, the detection unit 102 determines that the window image 201contains the object (face) which is the detection target. On the otherhand, if any weak classifier determines NG during the processing, thedetection unit 102 cancels the subsequent processes, and determines thatthe window image 201 does not contain the object (face), which is thedetection target. In this way, N pieces of the weak classifiers of theweak classifiers 210 to 250 are combined to construct a robustclassifier as a whole.

Next, the flow of the overall processing will be described. FIG. 3illustrates processing of detecting an object by applying a detectionwindow to an image. A detection window 611 is applied to image data(hereinafter also referred to as just “image”) 601. The detection window611 is the same as the window image 201 illustrated in FIG. 2. Duringthe detection processing, the detection unit 102 applies the detectionalgorithm illustrated in FIG. 2 to each position while moving the window611 in the image 601 so as to partially overlap another window. Windows612 and 613 illustrated in FIG. 3 are examples of the moved window.Especially, the window 613 is an example of a window located on theposition of a face which is the detection target.

The detection algorithm illustrated in FIG. 2 needs to have a highgeneralization property. This is because an image that is a detectiontarget during the execution of the processing, especially, a detectiontarget or an object cannot be viewed in advance, and therefore thealgorithm has to process an unknown image pattern. Therefore, thedetection algorithm is designed so as to be able to detect a detectiontarget (face) even if a change occurs to some degree in the size,position, and shape relative to the detection window. However, it isextremely difficult to design the detection algorithm so as to be ableto detect even an object significantly different from the size,position, and shape of the target expected by the algorithm.

FIG. 4 illustrates states in which a size of a face as the detectiontarget varies. In an image 710, objects 701 to 703 are a person indifferent sizes in the image. In the image 710, the object 701 is smallin size because the object 701 is positioned away from an imagingposition. On the contrary, the object 703 is large in size because theobject 703 is positioned relatively near to the imaging position.Further, rectangles 711 to 713 are rectangles roughly surrounding therespective face regions of the objects 701 to 703, and the sizes thereofare different. Therefore, a method illustrated in FIG. 5 is employed asa method for respectively detecting detection targets in various sizesdepending on, for example, how they are imaged in this way.

FIG. 5 illustrates a processing procedure of the detection processing.The image 710 is the same as the image illustrated in FIG. 4, and is adetection target image input to the system. The detection unit 102applies detection processing 820 (the algorithm illustrated in FIG. 2)thereto. The image magnification varying unit 110 outputs reduced images801 and 802 by reducing the image 710. The detection unit 102 performsdetection by applying the detection processing 820 to these reducedimages 801 and 802 as well.

Detection windows 810 to 812 in the respective images are the same insize. However, the image resolution of the processing target isdifferent among them, and the detection window 810 matches a face in arelatively small size. On the contrary, the detection window 812 matchesa face in a relatively large size. Further, the detection window 811matches a face in an intermediate size between the detection window 810and the detection window 812. In other words, in the image 710 at highresolution, the face in a small size is detected. In the image 802 atlow resolution, the face in a large size is detected. In the image 801at intermediate resolution, the face in an intermediate size isdetected.

The method for reducing an image may be any known method. In addition,the magnification ratio at which the image is reduced may be determinedin advance. In the present exemplary embodiment, the image magnificationvarying unit 110 generates the reduced image 801 from the image 710, andgenerates the reduced image 802 from the reduced image 801. In the casewhere such a method is used, actually, the image reduction cannot avoidbeing performed in a discrete manner to some degree (for example, theimage is horizontally and vertically reduced by ½ or 1/√2). However,since the detection algorithm has the high generalization property asdescribed above, the detection target can be detected if the face ineach image has a somewhat different size from the size that thedetection algorithm expects originally.

On the other hand, a situation illustrated in FIG. 6 may occur. FIG. 6illustrates that the detection algorithm returns a plurality ofdetection results for a single object. The image 710, the reduced images801 and 802, and the detection window 811 are the same as thoseillustrated in FIG. 5. In the example illustrated in FIG. 5, thedetection window 811 matches the object's face in the reduced image 801.However, actually, since the algorithm has the generalization property,the same object may be detected in another image at different resolution(windows 911 and 913), or the object may be detected at a slightlyshifted position (window 912) in the image at the same resolution.

FIG. 7 illustrates a state in which the plurality of matching windowsare associated with one another in consideration of the magnificationratios of the images. The windows (regions) 811 and 911 to 913correspond to the windows indicated by the same reference numerals inFIG. 6. However, actually, there is only one object, so that theseregions 811 and 911 to 913 need to be integrated into one region. Thisprocessing is the integration processing by the integration processingunit 104.

The integration processing is processing for integrating the regions 811and 911 to 913 illustrated in FIG. 7 into one region. The integrationprocessing unit 104, for example, calculates the centers (the centroids)of the respective regions 811 and 911 to 913, and determines that theyare regions detecting a same object if a difference among them is equalto or smaller than a threshold value to integrate them. The thresholdvalue may be determined in advance. Further, the integration processingunit 104 integrates the regions by, for example, calculating averages offour vertexes of the rectangle among the respective regions. Theintegration processing method described herein is merely an example, andthe integration processing method may be any method capable ofgenerating one region from the regions 811 and 911 to 913 illustrated inFIG. 7.

For the integration processing, the detection unit 102 once stores adetection result from the detection processing applied to one image(hereinafter referred to as “intermediate result” to distinguish it fromthe result after the integration). Then, the integration processing unit104 analyzes a relationship among the detection results (the distanceamong the centers of the intermediate results in the above-describedexample) to make a determination. Therefore, the intermediate resultsneed to be temporarily stored. Accordingly, a storage area for storingthe intermediate results will be prepared, and it is difficult todetermine how large area to be prepared.

First, it is impossible to estimate how many intermediate results willbe detected in a detected image before actually detection. Next, atheoretical upper limit value is considered. For example, suppose thatthe resolution of an input image (the image 710 illustrated in FIG. 5)is approximately Video Graphic Array (VGA) (640×480 pixels). Assumingthat the window has a size of 20×20 pixels, and can be set to be shiftedfrom the adjacent window by one pixel, and the image is vertically andhorizontally reduced by 1/√2 (the number of pixels is reduced by ½), theapproximate number of windows can be calculated by the followingequation.

${{THE}\mspace{14mu} {APPROXIMATE}\mspace{14mu} {NUMBER}\mspace{14mu} {OF}\mspace{14mu} {WINDOWS}} = {\frac{\left( {640 - 20} \right) \times \left( {480 - 20} \right)}{1 - \frac{1}{2}} \cong 570000}$

(from the sum of infinite series)Actually, the image cannot be reduced infinitely, so that the actualnumber of the window is smaller than the above calculated number, butyet approximately 500000 or more windows will be processed.

If the resolution of an input image is Quarter Video Graphic Array(QVGA) (320×240 pixels), the number of windows acquired from the similarcalculation is approximately 130000. Further, if the processing isthinned out and the windows are set so as to be shifted from theadjacent window by one pixel vertically and horizontally, the resultwill be a quarter of the prior case, so that approximately 30000 windowscan be set. Alternatively, if the processing is further thinned out andthe windows are set so as to be shifted from the adjacent window bythree pixels vertically and horizontally, the result will beone-sixteenth, so that less than 10000 windows can be set. Therefore, ifthe processing is performed with a memory corresponding to thetheoretical upper limit value prepared, for example, assuming that anarea of 10 byte is required to store necessary information for oneintermediate result, a memory of 100000 byte 100 KB is required to storeapproximately 10000 intermediate results.

However, when the detection processing was actually performed withapproximately 10000 windows set to images (an input image and all of thereduced images thereof), the result was that many images output at mostapproximately 100 to 200 intermediate results. In other words, manyimages use only approximately 1 to 2 percent of the storage areaprepared based on the theoretical upper limit value.

A memory of 100 KB may not become a big problem when the detectionprocessing is realized in, for example, a personal computer. However,for example, when the detection processing is realized by softwareembedded in a device or a circuit, a memory of 100 KB is a largecapacity as a memory (static random access memory (SRAM)) within anembedded device, and affects the price of the device. In addition, it isobvious that preparing a memory corresponding to the theoretical upperlimit value is extremely redundant in consideration of the fact that,normally, only approximately 1 to 2 percent thereof is used. Therefore,it is desirable to realize the detection processing with use of a memoryof an appropriate size (for example, a size capable of storingapproximately 200 results according to the experiment). However, ashortage of the storage area may occur depending on the content of animage.

The shortage of the storage area can be solved in the following manner.FIG. 8 illustrates an image containing faces having various sizes. Theimage illustrated in FIG. 8 contains an object 401 (an object image in asmall size) to an object 405 (an object image in a large size). On theother hands, FIG. 9 illustrates examples of intermediate resultsacquired from detection for the respective objects. Actually,intermediate results having different positions and sizes may bedetected with respect to intermediate results 501 to 505 as describedabove, however such results are omitted in FIG. 9 for simplification ofthe description.

FIG. 10 illustrates processing applied to the image illustrated in FIG.8. The elements in FIG. 10 same as those in FIGS. 8, 9, and 5 areindicated by the same reference numerals, and therefore descriptionsthereof will be omitted here. First, the detection unit 102 performs thedetection processing 820 on an input image 1101, and outputs, forexample, intermediate results 501 and 502. Next, the detection unit 102performs the detection processing 820 on a reduced image 1102, andoutputs an intermediate result 503. Further, the detection unit 102performs the detection processing 820 on a reduced image 1103, andoutputs intermediate results 504 and 505.

FIGS. 11A to 11C illustrate the state of the intermediate result storageunit 103. For simplification of description, the present exemplaryembodiment is described assuming that the intermediate result storageunit 103 includes intermediate result storage areas 1201 to 1203corresponding to 3 results. The case in which five results, namely theintermediate results 501 to 505 are output to the intermediate resultstorage areas is described as illustrated in FIG. 10.

First, the intermediate results 501 and 502 are output from the inputimage 1101, and they are stored in the intermediate result storage areas1201 and 1202, respectively (FIG. 11A). Subsequently, the detectionresult 503 is output from the reduced image 1102, and stored in theintermediate result storage area 1203. As a result, the intermediateresult storage unit 103 is brought into a buffer full state (FIG. 11B).When the subsequent intermediate results 504 and 505 are output in thebuffer full state, the simplest way to handle this case is to discardthe intermediate results 504 and 505 without storing them into theintermediate result storage unit 103. FIG. 12A illustrates an example ofthe detection result in this case.

Referring to the results illustrated in FIG. 12A, this looks like as ifit has failed to detect the central two people whose images arerelatively large. Empirically, a larger object can be considered moreimportant than a smaller object in a captured image. As viewed from adifferent perspective, it is considered that the faces of the centraltwo people have higher resolution, and are relatively easy to bedetected. Nevertheless, the result illustrated in FIG. 12A looks like asif it has failed to detect faces in the relatively large size.Therefore, this method provides an output as if it has failed to detect(empirically) important and supposedly easily detectable objects.

Instead, the processing method according to the present exemplaryembodiment is performed as follows. In the processing method accordingto the present exemplary embodiment, the detection unit 102 circularlyuses the intermediate result storage areas 1201 to 1203 of theintermediate result storage unit 103 and writes the results therein(regardless of whether data has been already stored in that area). FIG.11C illustrates the state of the intermediate result storage unit 103when the detection unit 102 uses the intermediate result storage area1201 as the storage area next to the intermediate result storage area1203 to store the intermediate result 504 therein, and similarly, storesthe detection result 505 in the intermediate result storage area 1202.For example, if there are further intermediate results thereafter, thedetection unit 102 circularly stores them into the intermediate resultstorage areas 1203 and 1201 in this order subsequently.

FIG. 13 is a flowchart illustrating the processing for circularly usingthe intermediate result storage areas 1201 to 1203 to store theintermediate results therein according to the first exemplaryembodiment. A variable “i” in the flowchart in FIG. 13 is initialized tozero by the processing other than that in FIG. 13 when the imageprocessing apparatus is started up or an input image as a detectiontarget is changed.

First, in step S1301, the detection unit 102 stores an output detectionresult (intermediate result) in an i-th buffer. In the exampleillustrated in FIGS. 11A to 11C, the intermediate result storage area1201 is a 0-th buffer. The intermediate result storage area 1202 is afirst buffer. The intermediate result storage area 1203 is a secondbuffer. The storage of the data contains information indicating thecoordinates and likelihood of the rectangle the detection result. Atthis time, the coordinates of the rectangle may be the coordinatesacquired by converting the rectangle of the corresponding detectionresult into data in the coordinate system of the input image 1101.Alternatively, the stored data can be configured to contain informationindicating the coordinates in the coordinate system of the detectedimage and which image the result is output from (“IMAGE” in theintermediate result storage areas 1201 to 1203 illustrated in FIGS. 11Ato 11C corresponds to the information for identifying the image). In thelatter case, the integration processing unit 104 converts thecoordinates in the detection result into data in the coordinate systemof the input image 1101 when performing integration processing.

Subsequently, in step S1302, the detection unit 102 increments thevariable i by one.

Then, in step S1303, the detection unit 102 compares the variable i andthe buffer size N. If i is equal to or larger than N (YES in stepS1303), the processing proceeds to step S1304. If not (NO in stepS1303), the processing is ended. In step S1304, the detection unit 102resets the value of the variable to zero. This is because i≧N issatisfied when the detection processing reaches the final buffer, sothat, in this case, setting the variable i to zero in step S1304 enablesthe intermediate result storage areas 1201 to 1203 of the intermediateresult storage unit 103 to be circularly used.

FIG. 12B illustrates an example of the detection result when theintermediate result storage unit 103 is in the state illustrated in FIG.11C in the method according to the present exemplary embodiment. In FIG.12B, the detection results are displayed on two people at the centralposition who are supposed to be relatively important, instead of losingthe detection results corresponding to the upper right two people in asmall size. Therefore, the comparison with the results in FIG. 12Areveals that the results in FIG. 12B are more appropriate. As describedabove, during the detection processing, the present exemplary embodimentcauses an image at higher resolution to be processed first so that asmaller detection result is detected first. Then, the present exemplaryembodiment circularly uses buffer areas for storing intermediateresults, thereby allowing a larger detection result to be prioritized ina case where the number of output intermediate results exceeds thepre-prepared buffer areas. Accordingly, an excellent result can beacquired.

The configuration illustrated in FIG. 1 includes the image storage unit101, the intermediate result storage unit 103, and the result storageunit 105 as separate storage areas. This is only because data storedtherein is different. Therefore, any of them may be constituted by asingle storage unit (for example, a memory, a resister, or a diskapparatus). Especially, a modification including the intermediate resultstorage unit 103 and the result storage unit 105 as a single storageunit may be effective in reducing the required number of memories.

Further, the configuration illustrated in FIG. 1 can be modified invarious manners. FIG. 14 illustrates an example of a hardwareconfiguration of the image processing apparatus according to the firstexemplary embodiment. The configuration illustrated in FIG. 14 isconfigured in such a manner that all of instructions and detectionresults (intermediate results) of the detection unit 102 are oncetransferred to the integration processing unit 104, and the integrationprocessing unit 104 stores the detection results into the intermediateresult storage unit 103. Such a modification is also within the scope ofthe present exemplary embodiment. In this case, the processingillustrated in FIG. 13 can be realized by being performed by theintegration processing unit 104.

Further, as is the case with the configuration example illustrated inFIG. 1, in the configuration illustrated in FIG. 14, the intermediateresult storage unit 103 and the result storage unit 105 may beconstituted by a single storage unit.

Alternatively, the intermediate result storage unit 103 may beduplicated to allow the detection unit 102 and the integrationprocessing unit 104 to operate in a pipeline manner. In this case, atthe same time that the integration processing unit 104 performs theintegration processing on the detection results (intermediate results)from the first image, the detection unit 102 can perform the detectionprocessing on the second image.

The processing according to the flowchart illustrated in FIG. 13 may bereplaced with the processing according to the flowchart illustrated inFIG. 15. FIG. 15 is a (second) flowchart illustrating processing forstoring intermediate results by circularly using the intermediate resultstorage areas 1201 to 1203 according to the first exemplary embodiment.The processing in step S1301 illustrated in FIG. 15 is the same as thatin step S1301 in the flowchart illustrated in FIG. 13, however thedetection unit 102 performs calculation in step S1401, instead of theprocessing in step S1302 and the steps thereafter. The equation in stepS1401 includes “mod”, which indicates the residue arithmetic. Inaddition, especially, if the value of N is a power of 2 and equal to orlarger than 2, instead of the calculation in step S1401, the detectionunit 102 may perform calculation by the following equation:

i=(i+1) & (N−1) (“&” represents bitwise AND)

The method using the equation including “mod” is effective when, forexample, a performance reduction due to the conditional branchingexceeds the cost required for the calculation of the residue arithmetic.Further, the method using bitwise AND can realize the processing withthe bit AND which requires less cost for the calculation than the “mod”calculation, and therefore is effective when the specified condition(the value of N is a power of 2 and equal to or larger than 2) issatisfied.

As described above, the image processing apparatus according to thefirst exemplary embodiment circularly uses the buffer areas for storingintermediate results during the detection processing for detecting atarget using fixed-sized window processing while reducing an image. As aresult, it is possible to acquire an excellent result even when thenumber of output intermediate results exceeds the number of bufferareas.

The present exemplary embodiment is described based on an example inwhich the respective units are implemented as hardware as illustrated inFIG. 1. However, as illustrated in FIG. 16, the respective units may berealized as software in such a manner that a central processing unit(CPU) 1 executes a program stored in a memory 2 to cause it to functionas the respective units. In this case, a medium (for example, a memory)for storing the program is one exemplary embodiment according to thepresent exemplary embodiment. In the configuration illustrated in FIG.16, the intermediate result storage unit 103, the result storage unit105, and the image storage unit 101 are realized in the memory 2,however this configuration does not limit the present exemplaryembodiment. As described above, the respective storage units may beconstituted by respectively different storage media (for example, amemory, a resister, and a disk apparatus).

Next, a second exemplary embodiment will be described. Regarding thesecond exemplary embodiment, only a difference from the first exemplaryembodiment will be described. FIG. 1 of the first exemplary embodimentillustrates an example including the image magnification varying unit110, however, this is merely an example. The image processing apparatusaccording to the present exemplary embodiment, the detection unit 102may be configured to process an image at high resolution first andprocesses an image at low resolution after that. Thus, for example, ifmagnified images at various degrees of resolution can be prepared in theimage storage unit 101 in advance, the image magnification varying unit110 may be omitted. For example, image data, which is coded accordingto, for example, the known Joint Photographic Experts Group (JPEG) 2000standard coding method, can be decoded into images at various degrees ofresolution. Therefore, the image processing apparatus may be configuredin such a manner that an image is sequentially decoded by a decodingprocessing unit (or a decoding processing apparatus) so as to begradually output as images from an image at high resolution to an imageat low resolution, stored into the image storage unit 101, and thenprocessed by the detection unit 102.

Next, a third exemplary embodiment will be described. Regarding thethird exemplary embodiment, only a difference from the first exemplaryembodiment will be described. FIG. 17 illustrates an example of thehardware configuration of an image processing apparatus according to thethird exemplary embodiment. The configuration illustrated in FIG. 17will be described by comparison with the configuration illustrated inFIG. 1. In FIG. 17, similar elements to those illustrated in FIG. 1 areindicated by the same reference numerals, and descriptions thereof willbe omitted.

Referring to FIG. 17, image storage units 1601 to 1603 are configured ina similar manner to the image storage unit 101 illustrated in FIG. 1.However, the image storage units 1601 to 1603 store different imagesfrom one another. In other words, for example, the images 710, 801, and802 illustrated in FIG. 5 are stored in the image storage units 1601,1602, and 1603, respectively. It is noted that the image storage units1601 to 1603 are illustrates as different blocks because they can beseparated conceptually, and they may be embodied by a single storagemedium physically (for example, a known memory, resister, and diskapparatus). The detection units 1611 to 1613 are configured in a similarmanner to the detection unit 102 illustrated in FIG. 1. An onlydifference is that the detection units 1611 to 1613 receive input imagesfrom the image storage units 1601 to 1603, respectively. A recordingmanagement unit 1621 stores detection results (intermediate results)output from the detection units 1611 to 1613 into the intermediateresult storage unit 103.

A processing flow by the recording management unit 1621 at this timewill be described with reference to FIG. 18. FIG. 18 is a flowchartillustrating processing performed when the recording management unit1621 receives a detection result (intermediate result) from any of thedetection units 1611 to 1613. A variable M in FIG. 18 represents thenumber of received intermediate results, and is initialized to zero bythe processing other than that in this flowchart when the imageprocessing apparatus is started up or an input image as a detectiontarget is changed, as similar to the variable i in FIG. 13 according tothe first exemplary embodiment.

First, in step S1701, the recording management unit 1621 compares thevariable M and the variable N. The variable N represents the number ofbuffers in the intermediate result storage unit 103, as similar to thevariable N in the flowchart in FIG. 13. The expression in step S1701 istrue when the intermediate result storage unit 103 is not in a bufferfull state, and in that case (YES in step S1701), the recordingmanagement unit 1621 advances the processing to step S1702. If theexpression in step S1701 is false (NO in step S1701), the recordingmanagement unit 1621 advances the processing to step S1703.

In step S1702, the recording management unit 1621 stores the currentlyreceived intermediate result in an M-th buffer. Then, in step S1710, therecording management unit 1621 increments the value of the variable M byone. Then, the processing is ended.

On the other hand, if the determination in step S1701 is false (NO instep S1701), the recording management unit 1621 advances the processingto step S1703. In step S1703, variables to be used in the followingsteps are initialized. Thus, the recording management unit 1621initializes a variable j to zero and a variable k to one.

The processing in step S1704 is for determining whether the loopconstituted by subsequent steps S1705 to S1707 can be ended. If thevariable k is equal to or larger than the variable N (NO in step S1704),the recording management unit 1621 ends the loop, and advances theprocessing to step S1708. If the variable k is smaller than N (YES instep S1704), the recording management unit 1621 advances the processingto step S1705 at the loop side.

In step S1705, the recording management unit 1621 compares the imageresolution of the result stored in a k-th buffer and the imageresolution of the result stored in a j-th buffer. In the presentexemplary embodiment, the image resolution of the result can bedetermined by, for example, determining from which detection unit theintermediate result is received among the detection units 1611 to 1603.In addition, as illustrated in FIG. 11 of the first exemplaryembodiment, the image resolution of the result is stored in theintermediate result storage areas 1201 to 1203. If “IMAGE” is containedin the information stored in the areas 1201 to 1203, the recordingmanagement unit 1621 can recognize the image resolution of the resultfrom the information. Alternatively, the recording management unit 1621can recognize the image resolution of the image by checking the size ofa rectangle if the coordinates of the rectangle are expressed by thecoordinate system after conversion into the coordinate system of theinput image 1101. The larger rectangle indicates a detection result(intermediate result) of an image at lower resolution.

If the determination in step S1705 is true (YES in step S1705), therecording management unit 1621 advances the processing to step S1706. Instep S1706, the recording management unit 1621 assigns the variable k tothe variable j. On the other hand, if the determination in step S1705 isfalse (NO in step S1705), the recording management unit 1621 skips stepS1706 to advance the processing to step S1707. In step S1707, therecording management unit 1621 increments the value of the variable k byone, and returns the processing to step S1704.

The loop processing from steps S1704 to S1707 is processing forsearching for a detection result corresponding to an image at thehighest resolution among the received intermediate results, and settingthe variable j to a value indicating the location of that detectionresult in the buffer.

If the determination in step S1704 is false (NO in step S1704), and theloop processing in steps S1704 to S1707 is ended, the recordingmanagement unit 1621 advances the processing to step S1708.

In step S1708, the recording management unit 1621 compares the (highest)image resolution searched for in the previous loop with the imageresolution of the currently received intermediate result. If the imageresolution of the received intermediate result is lower than the(highest) image resolution in the buffer (YES in step S1708), therecording management unit 1621 advances the processing to step S1709.

In step S1709, the recording management unit 1621 overwrites thedetected stored intermediate result with the received detection result,and then advances the processing to step S1710. If not (NO in stepS1708), the recording management unit 1621 advances the processing tostep S1710. In the latter case, the received intermediate result isdiscarded.

In other words, the recording management unit 1621 records the receivedintermediate result in an unoccupied space if the intermediate resultstorage unit 103 is not in the buffer full state. On the contrary, ifthe intermediate result storage unit 103 is in the buffer full state,the recording management unit 1621 searches for an intermediate resultcorresponding to an image at the highest resolution among the storeddetection results. Then, the recording management unit 1621 compares theresolution of the image of the detected intermediate result with theresolution of the image of the received intermediate result, and storesthe result of the image at lower resolution. The flowchart in FIG. 18 isan example of processing therefor, and a modification using processingthat works similarly is included in the scope of the present exemplaryembodiment.

The present exemplary embodiment is described based on an example inwhich the image storage units 1601 to 1603 store the images 710, 801,and 802 illustrated in FIG. 5, respectively, and the detection units1611 to 1613 perform the detection processing on the images 710, 801,and 802, respectively. However, actually, the processing loads on thedetection units 1611 to 1613 may be uneven because of the difference inthe resolution of the images. In this case, for example, the imageprocessing apparatus may be configured in such a manner that the images710, 801, and 802 are appropriately divided to be distributed to theimage storage units 1601 to 1613, and the detection units 1611 to 1613perform the detection processing thereon, respectively.

Further, the image processing apparatus may be configured in such amanner that the identification information (or the information about theresolution) of images set as detection targets of intermediate resultsoutput from the detection units 1611 to 1613 is provided to therecording management unit 1621. It is desirable that the imageprocessing apparatus is configured to include a control unit (notillustrated in FIG. 17) for controlling image division and distribution,and transmit the identification information of images from the controlunit to the recording management unit 1621.

As described above, the first exemplary embodiment can be modified, forexample, as illustrated in FIG. 14. Further, the present exemplaryembodiment can be also modified in a similar manner. In this case, theimage processing apparatus may be configured in such a manner thatoutputs from the plurality of detection units 1611 to 1613 are firstreceived by the integration processing unit 104, and the recordingmanagement unit 1621 outputs the intermediate results to theintermediate result storage unit 103 in response to the outputs from theintegration processing unit 104. Alternatively, the image processingapparatus may be configured in such a manner that the integrationprocessing unit 104 carries out the function of the recording managementunit 1621.

The present exemplary embodiment is described as an example in which theorder relationship among the intermediate results output from thedetection units 1611 to 1613 is different from the first exemplaryembodiment. The method according to the present exemplary embodiment canprovide an advantageous effect of facilitating parallelization ofprocessing and distribution of processing by providing, for example, theplurality of detection units 1611 to 1613, compared to the firstexemplary embodiment.

Next, a fourth exemplary embodiment will be described. Regarding thefourth exemplary embodiment, only a difference from the first and thirdexemplary embodiments will be described. The first and third exemplaryembodiments are described based on an example in which the magnificationratio of an input image is changed. In the present exemplary embodiment,it is described a method for performing the detection processing whilevarying the magnification of the window size (in other words, performingthe detection processing while changing the window size).

An image processing apparatus according to the present exemplaryembodiment refers to a part or all of a window image during thedetection processing, and performs determination processing thereon withuse of a predetermined coefficient. The present exemplary embodimentsequentially doubles, triples, and so on, the window size and thereference position and range in the window, and thereby can acquire asimilar effect when the magnification ratio of an input image is reducedto one-second, one-third, and so on. In this case, the image processingapparatus according to the present exemplary embodiment can perform thedetection processing using, for example, the magnification ratio or sizeof the window, instead of the information about the resolution (or theidentification information) of an image set as a detection target usedin the first and third exemplary embodiments.

According to the first and third exemplary embodiments, the imageprocessing apparatus is configured to discard a detection result of adetection target image at high resolution and save a detection result ofa detection target image at lower resolution. This configuration isintended to save a detection result corresponding to a larger detectiontarget as described above with reference to FIGS. 4 and 5. Accordingly,the image processing apparatus according to the present exemplaryembodiment is configured to discard a detection result corresponding toa window with a smaller magnification ratio, and save a detection resultcorresponding to a window with a larger magnification ratio. To achievethis operation, the image processing apparatus according to the presentexemplary embodiment records the magnification ratio (or informationallowing identification of it), instead of the “IMAGE” information, whenintermediate results are stored as illustrated in FIGS. 11A to 11C.

The image processing apparatus according to the present exemplaryembodiment performs the detection processing on a whole area of an inputimage with a predetermined minimally-sized window, and then increasesthe window size to perform the detection processing again on the wholearea of the input image. Then, the image processing apparatus accordingto the present exemplary embodiment further repeats the detectionprocessing with a magnified-sized window. The image processing apparatusaccording to the present exemplary embodiment is configured in this way,and can be also realized by the processing illustrated in FIG. 13 orFIG. 15 of the first exemplary embodiment. As described above, accordingto the present exemplary embodiment, it is also possible to acquire anexcellent result even when the number of output intermediate resultsexceeds the number of buffer areas.

Further, the present invention can be also implemented by performing thefollowing processing. More specifically, software (a program) forrealizing the functions of the above exemplary embodiments is suppliedto a system or an apparatus via a network or various storage media, anda computer (or a CPU or a micro processing unit (MPU)) of the system orthe apparatus reads and executes the program.

As described above, according to each of the exemplary embodiments, itis possible to realize object detection processing with use of a storagecapacity of a smaller size than a fixed predetermined size.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims priority from Japanese Patent Application No.2011-139687 filed Jun. 23, 2011, which is hereby incorporated byreference herein in its entirety.

1. An image processing apparatus comprising: a detection unit configuredto scan an input image and each of images at different resolutions,which are generated from the input image, by a predetermined-sizedwindow to detect an object in the image; a storage unit configured tostore a detection result of the detection unit; and a control unitconfigured to, if there is no free space in the storage unit to store anew detection result of the detection unit, store the new detectionresult instead of a detection result of an image at higher resolutionthan resolution of an image from which the new detection result isacquired.
 2. The image processing apparatus according to claim 1,wherein the storage unit includes a predetermined number of storageareas, the detection unit scans the input image and each of the imagesat the plurality of degrees of resolution, which are generated from theinput image, by the predetermined-sized window, starting from an imageat high resolution in descending order of resolution to detect theobject in the image, and the control unit sequentially stores thedetection results of the detection unit in the storage areas, and in acase where there is no free space in the storage area, overwrites thedetection result stored in the storage area with the new detectionresult, starting from the storage area in which the detection result isstored first, in the order in which the detection results are stored inthe storage areas.
 3. The image processing apparatus according to claim1, further comprising an integration unit configured to integrate aplurality of detection results stored in the storage unit.
 4. An imageprocessing apparatus comprising: a detection unit configured to scan aninput image by windows having a plurality of sizes to detect an objectin the image; a storage unit configured to store a detection result ofthe detection unit; and a control unit configured to, if there is nofree space in the storage unit to store a new detection result of thedetection unit, store the new detection result instead of a detectionresult by a window of a smaller size than a size of a window by whichthe new detection result is acquired.
 5. The image processing apparatusaccording to claim 4, wherein the storage unit includes a predeterminednumber of storage areas, the detection unit scans the input image by thewindows having the sequentially increased sizes to detect the object inthe image, and the control unit sequentially stores the detectionresults of the detection unit in the storage areas, and in a case wherethere is no free space in the storage area, overwrites the detectionresult stored in the storage area with the new detection result,starting from the storage area in which the detection result is storedfirst, in the order in which the detection results are stored in thestorage areas.
 6. The image processing apparatus according to claim 4,further comprising an integration unit configured to integrate aplurality of detection results stored in the storage unit.
 7. A methodfor processing an image, the method comprising: scanning an input imageand each of images at different resolutions, which are generated fromthe input image, by a predetermined-sized window to detect an object inthe image; and storing a new detection result instead of a detectionresult of an image at higher resolution than resolution of an image fromwhich the new detection result is acquired, if there is no free space ina storage unit to store the new detection result of the detection.
 8. Amethod for processing an image, the method comprising: scanning an inputimage by windows having a plurality of sizes to detect an object in theimage; and storing a new detection result instead of a detection resultby a window of a smaller size than a size of a window by which the newdetection result is acquired, if there is no free space in a storageunit to store the new detection result of the detection.
 9. Anon-transitory computer-readable storage medium storing a programcapable of causing a computer to execute a method according to claim 7.10. A non-transitory computer-readable storage medium storing a programcapable of causing a computer to execute a method according to claim 8.